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Background / Context: 

Description of prior research and its intellectual context. 


Methods for meta- analysis have made great strides over the past 30 years. The focus of this work 
has been primarily on synthesizing univariate and bivariate effect size estimates. Increasingly, 
however, multivariate models such as multivariate linear models, factor analyses, and latent trait 
models are used to describe relationships in the social sciences and in medicine. Meta-analytic 
methods to combine effect sizes from these kinds of models are often cumbersome to implement 
or nonexistent. 

Multiple regression is one of the most commonly used statistical methods in the social sciences. 
Because of the multivariate nature of multiple regression coefficients, they have historically been 
very difficult to combine with meta- analysis. Becker and Wu (2007) outline three key difficulties 
in combining multiple regression slope estimates. First, all model outcomes must be measured on 
a common scale. Second, the slope estimate of interest (focal slope) is measured on a common 
scale across studies. Finally, each study estimates the partial relationship between the focal slope 
and the outcome using the model (i.e. includes an identical set of additional predictors). 
Maintaining these assumptions in any given synthesis will almost always be impossible. 

Purpose / Objective / Research Question / Focus of Study: 

Description of the focus of the research. 


The purpose of this study was to explore the use of robust variance estimation for combining 
commonly specified multiple regression models and for combining sample-dependent focal slope 
estimates from diversely specified models. The proposed estimator obviates traditionally 
required information about the covariance structure of the dependent effect size estimates, 
making it a potentially flexible method for conducing meta-analyses of regression estimates. 

A series of Monte Carlo simulations were conducted to explore the performance of the robust 
variance estimator under different meta-analytic conditions. 

Setting: 

Description of the research location. 

(May not be applicable for Methods submissions) 

Not applicable. 

Population / Participants / Subjects: 

Description of the participants in the study: who, how many, key features, or characteristics. 

(May not be applicable for Methods submissions) 

Not applicable. 

Intervention / Program / Practice: 

Description of the intervention, program, or practice, including details of administration and duration. 
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(May not be applicable for Methods submissions) 


Not applicable. 

Significance / Novelty of study: 

Description of what is missing in previous work and the contribution the study makes. 


Researchers across different fields are clearly eager to make use of multiple regression estimates 
in meta-analytic fashion. Some of the methods in use are intuitive, others are considerably more 
complex. Many of the methods, however, fail to address the fundamental problems of combining 
multiple regression estimates: collinearity among model predictors and diverse model 
specification. Other methods, such as generalized least squares and individual participant data 
meta-analytic techniques are promising but often the data needed to use those methods are 
unavailable. A flexible alternative is needed to meta-analyze multiple regression estimates. 

Statistical, Measurement, or Econometric Model: 

Description of the proposed new methods or novel applications of existing methods. 


Because the covariance structure of correlated effect size estimates is almost never report (or 
even explored) in primary research. Hedges, Tipton, and Johnson (2010) discuss in detail the use 
of robust standard errors in meta-regression with dependent effect size estimates. Their approach 
has been used with standardized mean differences and effect size estimates for binary outcomes. 
It has not yet been applied to multiple regression studies. 

This study uses the robust variance estimator to combine commonly specified models across a 
pool of study samples and to combine multiple, sample-dependent, focal slope estimates derived 
from diversely specified models across study samples. More details of the estimator are provided 
in Appendix B. 

Usefulness / Applicability of Method: 

Demonstration of the usefulness of the proposed methods using hypothetical or real data. 


Two sets of simulations studies were conducted. First, a series of focal slope meta-analyses were 
conducted. Here, a regression model was specified for each study sample (m), a common slope 
estimate (e.g. x\) was extracted and the inclusion of additional study covariates in that model was 
recorded. As such, the vector of slope estimates to be combined with meta-analysis was derived 
from diversely specified models. And, each study sample estimated multiple (k) focal slopes. 
Second, a series of model-based meta-analyses were conducted. Here, m study samples were 
generated and a common model was specified for each sample. Each study sample, then, 
contributed a vector of slope estimates to the meta- analyses. The following parameter values 
were used in the simulations: n, is the primary study sample size; m, the number of study 
samples per meta-analysis; k, the number of slope estimates per study sample; p, the correlation 
among slope estimates; and /^, the proportion of slope estimate heterogeneity. For each 
approach, a total of 243 parameter combinations were used. For each combination 1000 meta- 
analyses were conducted. 
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Each meta- analysis was conducted using inverse-variance weight methods (e.g. Hedges & Olkin, 
1985) under fixed-effeets model assumptions. Robust standard errors were computed for each 
mean slope estimate for eaeh meta-analysis using simple fixed-effects weights. Parameter 
recovery and percent bias calculated for each meta-analysis to better understand the performance 
of the robust variance estimator. 

Research Design: 

Description of the research design (e.g., qualitative case study, quasi -experimental design, secondary analysis, 
analytic essay, randomized field trial). 

(May not be applicable for Methods submissions) 

Not applicable. 

Data Collection and Analysis: 

Description of the methods for collecting and analyzing data. 

(May not be applicable for Methods submissions) 

Not applicable. 

Findings / Results: 

Description of the main findings with specific details. 

(May not be applicable for Methods submissions) 

Overall, the robust standard errors recovered the parameter values specified in the simulated 
datasets. In the foeal slope approach, the mean estimates were biased, consistently 
underestimating the parameter value. However, after a bias adjustment the robust confidence 
intervals recovered the population value at or above the specified nominal (95 percent) value. In 
all conditions, the model-based approach demonstrated close to or above 95 percent parameter 
recovery. Efficiency was largely a function of the number of studies included in the meta- 
analysis, m. 

Conclusions: 

Description of conclusions, recommendations, and limitations based on findings. 


The results of the simulations for this approach were promising. The robust confidence intervals 
for each set of parameter combinations were close to nominal probability content in nearly all of 
the specified conditions. And, when they departed from nominal coverage, it was in the 
conservative direction. This was most notable when the ratio of studies m to the number of 
parameter estimates k was small. 

The robust variance estimator outlined by Hedges, Tipton, and Johnson (2010) provides a 
flexible method for combining multivariate results. The results of this study indicate that using 
robust variance estimation is a viable alternative to the methods outlined by Wu and Becker 
(2007). 
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Appendices 

Not included in page count. 


Appendix A. References 

References are to be in APA version 6 format. 
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Appendix B. Tables and Figures 

Not included in page count. 


Robust variance estimation and meta-analysis. Because the covariance structure of 
correlated effect size estimates is almost never report (or even explored) in primary 
research, Hedges, Tipton, and Johnson (2010) discuss in detail the use of robust standard 
errors in meta-regression with dependent effect size estimates. Their approach has been 
used with standardized mean differences and effect size estimates for binary outcomes. 
Their method may be applied to the synthesis of multiple regression slope estimates. 

The procedures to follow assume that dependencies occur through correlated error terms. 
As such, a correlated effects modeling approach is presented. Modeling multiple 
regression estimates takes the following linear model form 

b = + e , 

where b is a vector of n (from n = I to k) clustered partial effects in m studies (clusters), 
X is a design matrix of (kj x p) meta-regression covariates (e.g. model specification 
indicators), P is a vector of unknown meta-regression coefficients, and e is a vector of 
residuals. In the context of multiple regression estimates, b is a vector of stacked model 
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slope (and intercept) estimates. In the case of combining commonly specified models X may be 
an identity matrix and 
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The weighted least squares estimate of p is 








with variance 




where W is a weight matrix for the mth study such that W = diag(W i . . . Wm) such that Wj = 
diag(wji,...,Wjkj), £ is the mth study’s Cov(b) such that S = diag(Si, ... , S^). Hedges et al. show 
that every element of Sy does not need to be estimated but rather the average of linear 
combinations of Sy need be estimated, specifically 


Because study report information about Sy is almost never provided, an empirical estimate of Sy 
is needed. Hedges, Tipton, and Johnson (2010) show that the cross product of within-study 
residuals may be used as a crude approximation of Sy so that the robust variance estimator is 

' ' m \ /v— iwi 

x;w,.e;e;w,X; > XjWjXj 

where ejej is the jth study’s matrix of cross products of within-study residuals. Specifically, 

Bj = hj - XjP 


V- rx;w,x, IX 
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The asymptotic distribution of m as it approaches oo is 

V^(v)-nb-/?)^ iVp(o,ip), 

— n J? 

where V = mV . As m rises, b ~ N{p, V ). The estimated robust standard error of bj is then 


mvfj 
m — p' 



where vfj is the yth diagonal element of and is a finite population correction. A robust 
significance test of Pj follows a t distribution, specifically 



with m-p degrees of freedom. And, a robust confidence interval may be obtained using 

hj - Ca/2Sf < Pj < bj + Ca/2Sf , 

where c„ is the 1-a point on the t distribution with m-p degrees of freedom. 


Hedges, Tipton, and Johnson (2010) identify three important features of this estimator. First, and 
most importantly, the covariance structure of effect size estimates is not needed. Second, 
parameter estimates converge on the target parameter as the number of studies, not the number of 
cases within studies, rises. And, the authors show that accurate standard errors are produced with 
as few as 10 to 20 studies. Third, this estimator is unbiased for any set of weights. 


The authors show the calculation of the mean effect size estimate is 


~ ym ' 

and with identical weighting procedures used for within-study estimates, the robust variance 
estimate becomes 

where b is the jth cluster’s unweighted mean effect size estimate, bi is the weighted mean effect 
size estimate from, and wj is the weight assigned to the jth cluster of effect size estimates; 
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assumed to be uniform (i.e. common across within-study effect size estimates). When uniform 
weights are used is equal to (m-l)lrn^ times the value of the typical variance. And, the robust 
standard error estimate is equal to 1/m times the typical variance of hj when uniform weights 
are used. 


Weighting within-study estimate and cluster (study) total weights warrants consideration. 
Because between-study variance is likely, in most meta-analyses, to be much greater than within- 
study variance, an optimal weighting procedure is used. As such assigning each within-study 
estimate an equal weight would mean 


w, 


ij 


1 _ 1 

kjVj. 


where Vj^ is the average of the effect size estimate variances in study j. The authors note that 
little precision is lost with this approach so long as the covariance among within-study estimates 
is reasonably high. If the covariance structure of the estimates is known or estimable, it could be 
used to improve efficiency; however, the advantage of this approach is that it requires no 
information about the covariance structure to produce accurate standard errors. 

It is also possible to estimate between-study variances with this approach. Using fixed effects 
weights described above, the weighted residual sum of squares homogeneity statistic is 


Z TTi / ^ — iTTl \ / ^ — iTTl \ ^ \ \ 

b;w.b; - b;W,b,) x;w;b; j ■ 


and the estimated residual variance component is 


= 


Ob - m + tr [v ^ XjX,-)] + p * tr [v {zf., ^ [xjj^X; - xjx,-] ) 


Sf.i kjWj - tr[v(5;^i wfXjl ;X,)] 


where ]j is a kj x kj matrix of 1 ’s, p is the within study correlation of effects, and V is the inverse 
of the X'WX matrix specified as 


V = 



Knowing p may seem contradictory to this general approach, since we presume the covariance 
structure of dependent estimates is unknown. However, for this purpose, estimating a between 
study variance component, p may be roughly approximated without severe penalty so long as m 
is reasonably large. Ishak, Platt, Joseph, & Hanley (2008) simulated meta-regression analyses 
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under various scenarios that demonstrate this point, and Hedges, Tipton, and Johnson (2010) 
reinforce it in their work. 
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